Solutions to Instability Problems with Sequential Wrapper-based Approaches to Feature Selection

It is generally accepted that Wrapper approaches will outperform Filter-based approaches to feature selection, particularly in situations where an adequate amount of data is available. What is often overlooked is that Wrapper approaches can be unstable. For instance, different partitionings of the training data can result in different routes through the search space and thus in different feature subsets being selected. In this paper we illustrate examples of this problem and a solution based on the aggregation of several runs of a sequential search is suggested. This is essentially an ensemble solution to instability in feature subset selection and it does seem to stabilise the process.